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REMARKS 

Claims 46, 48, 50-54 and 56, 58-62 are currently pending in this 
application. Claims 46, 54, 61 and 62 have been amended. No new matter has 
been added to this application. 

Rejection of Claims 46, 48, 50-53, 61 and 62 under 35 U.S.C. § 101 

The Examiner has rejected claims 46, 48, 50-53, 61 and 62 under 35 
U.S.C. § 101 as being directed non-statutory subject matter. The Examiner 
contends that claims 54, 56 and 58-60 are directed towards nonfunctional 
descriptive material. Applicants respectfully traverse the rejection. 

As reflected in amended claims 46, 54, 61 and 61 , Applicants invention is 
directed to a computer implemented method for editing a video stream to 
combine augmented virtual images with video data from the video stream and 
displaying the augmented reality video. Contrary to the Examiner's assertion, 
Applicants invention produces a tangible result. Applicant's invention takes a 
video stream and combines it with an augmented image of an object that is 
contained in the video stream that has substantially the pose as that object. The 
resulting combined video is physically different from the original video stream 
which is then displayed. As such, Applicants respectfully submit that claims 46, 
48, 50-53, 61 and 62 are directed to statutory subject matter in accordance with 
35 U.S.C. § 101 and request that the rejection be withdrawn. 

Rejection of Claims 46, 52-54 and 60 under 35 U.S.C. § 102 (a) 

The Examiner has rejected claims 46, 52-54 and 60 under 35 U.S.C. § 
102 (a) as being anticipated by Kato ("Marker Tracking and HMD Calibration for 
a Video-Based Augmented Reality Conferencing Center 5 ', 20 October 1999). 
The Examiner contends that Kato teaches Applicants' invention as claimed. 
Applicants respectfully traverse the rejection. 
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The present invention is directed to a method for editing a video stream to 
combine augmented virtual images with video data. Video data comprising 
images of a moving model plane having markers is provided. The markers 
calibrate a camera to track the motion of the model plane. A pose of the moving 
model plane is determined according to the markers in the video data and 
calibration results. A three dimensional image data model of a product is 
provided in a pose corresponding to the pose of the moving model plane. An 
image correspondence between the camera calibration results associated with 
the moving model plane and the three-dimensional image data model is 
determined. A 3D model of the product based on the image correspondence is 
rendered. An augmented reality video is generated by superimposing the 
rendered 3D model of the product on the moving model plane in the video data. 
The augmented reality video is then displayed. 

Kato discloses an augmented reality conferencing system that uses the 
overlay of virtual images on the real world. Users are able to collaboratively view 
and interact with virtual objects using a shared virtual whiteboard. A user wears 
a head mounted display and can see video images from desktop users. The 
user also has a set of marked cards and a larger piece of paper with six letters 
around it on the outside. Each marked card corresponds to a different 
collaborator. The shared whiteboard is used so that the collaborators can share 
notes and diagrams. Essentially, Kato discloses a virtual conference space that 
is created by superimposing real videos of the conference attendees on top of 
the moving cards using the printed visual markers. Contrary to the present 
invention and as explicitly pointed out on in the text immediately under Figure 4, 
Kato acknowledges that their "application only supports virtual annotations 
aligned with the surface of the card, but we are working on adding support for 
shared 3D objects". Applicants respectfully submit that Kato does not teach or 
disclose the creation of an augmented reality video in which a 3D image of an 
object is augmented into a video stream. Because the object itself is a 3D object, 
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the pose of the object (i.e., the direction and angle of the object) is critical in 
creating an image model that reflects the object from the desired visual 
perspective. This problem is neither recognized nor addressed by Kato since 
Kato is deals with a 2D object. Furthermore, as exemplified above, Kato 
explicitly acknowledges that it does not have a solution for dealing with 3D 
objects. As such, Applicants respectfully submit that Kato does not teach or 
disclose Applicants' invention as claimed and request that the rejection of claims 
46, 52-54 and 60 under 35 U.S.C. § 102 (a) be withdrawn. 

Rejection of Claims 48 and 56 U.S.C. § 103 fa) 

The Examiner has rejected claims 48 and 56 under 35 U.S.C. § 103 (a) as 
being unpatentable over Kato in view of U.S. Patent No. 6,898,307 (Harrington). 
The Examiner correctly notes that Kato does not teach scaling the three 
dimensional image data model to the model plane according to the markers. The 
Examiner contends that Harrington discloses this limitation. The Examiner 
argues that it would have been obvious to one skilled in the art to modify Kato's 
system to scale the three dimensional virtual image as disclosed in Harrington. 
Applicants respectfully traverse the rejection. 

Harrington discloses a system for interacting with a real world item such 
as a piece of paper with a virtual display of imagery and/or text to give the illusion 
of holding a printed version of an electronic document. A blank piece of paper is 
disposed within a field of view and control of a user so that manipulation of the 
real item is intended to be communicative of user instructions. The item is 
selectively positionable by the user for purposes of communicating certain 
instructions to an applications program controlling the display. Essentially, the 
user can use the piece of paper as a computer monitor for the system. The 
tracked real object (i.e., the paper) is used the display by projecting the virtual 
view on the surface of it. 
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Unlike the present invention, Harrington deals with the 3D manipulation of 
a 2D object. As such, the positioning of the paper on the screen is an issue as 
well as its depth (whether type is larger or smaller). However Harrington does 
not need to concern itself with the pose of the object (i.e., the 3D spatial position 
of the object). The present invention is directed to the creation of an augmented 
reality video in which a 3D image of an object is augmented into a video stream. 
Because the object itself is a 3D object, the pose of the object (i.e., the direction 
and angle of the object) is critical in creating an image model that reflects the 
object from the desired visual perspective. This problem is neither recognized 
nor addressed by Harrington since Harrington is concerned with a 2D object. 
Applicants respectfully submit that Harrington is not concerned with the pose of 
an object. As such, Applicants submit that Harrington does not teach or disclose 
Kato is also directed to the manipulation of a two dimensional object. Applicant 
respectfully submits that neither Kato nor Harrington, whether taken alone or in 
combination, teaches or disclose creation of an augmented reality video in which 
a 3D image of an object is augmented into a video stream. Applicant requests 
that the rejection of claims 48 and 56 under 35 U.S.C. § 103 (a) be withdrawn. 

Rejection of Claims 61 and 62 under 35 U.S.C. § 103 (a) 

The Examiner has rejected claims 61 and 62 under 35 U.S.C. § 103 (a) as 
being unpatentable over Harrington in view of U.S. Patent No. 7,050,603 
(Rhoads). The Examiner correctly notes that Harrington does not teach a three 
dimensional image data model that represents an animation. The Examiner 
contends that Rhoads discloses watermarks that can be embedded in two 
dimensional image renderings or animated 3D graphical objects and that the 
embedded object can be composited with a video stream to form a video 
program. The Examiner argues that it would have been obvious to one skilled in 
the art to modify Harrington's system by using projected 3D animated objects as 
disclosed in Rhoads. Applicants respectfully traverse the rejection. 
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Applicants have amended independent claims 61 and 62 to recite the step 
of "creating a three dimensional image data model of a product in a pose 
corresponding to the pose of the moving model plane". Applicants respectfully 
submit that Harrington does not perform this step since Harrington is not 
concerned with the pose of an object. As such, Applicants submit that Harrington 
does not teach or disclose Applicants' invention as recited in independent claims 
61 and 62. 

Rhoads discloses using watermarks in video signals to associate video 
objects in a video sequence with object specific actions or information. 
Applicants respectfully submit that adding a watermark as is disclosed in Rhoads 
to the Harrington system would not teach or disclose Applicants' invention. 
Applicant's invention is directed to the editing of an augmented reality video in 
which a 3D object is augmented into a video stream. The 3D object can be 
displayed from various perspectives and angles so that certain features of the 
object are highlighted as recited in amended claims 61 and 62. As such, the 
pose of the 3D object is critical in augmented the appropriate view of the 3D 
object into the video stream. Watermarks are typically used to encode invisible 
or hard to detect information into a video or image. That information can be 
extracted with special processing of the video/image with corresponding 
computational algorithms. In contrast, the present invention attaches a piece of 
information, such as a website address, to the video created so that it can be 
easily retrieved by the viewer by clicking on the video display. 

As discussed above, Harrington is directed to displaying a paper 
containing information into an electronic image so that user instructions can be 
displayed. Harrington does disclose displaying a 3D object in different poses to 
highlight different perspectives of the 3D object. It makes even less sense for 
Harrington to disclose an object that is animated. Applicant respectfully submits 
that neither Harrington nor Rhoads, whether taken alone or in combination, 
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teaches or disclose Applicant's invention as claimed. Applicants request that the 
rejection of claims 61 and 62 under 35 U.S.C. § 103 (a) be withdrawn. 

Rejection of Claims 50, 51. 58 and 59 under 35 U.S.C. § 103 fa) 

The Examiner has rejected claims 50, 51 , 58 and 59 under 35 U.S.C. § 
103 (a) as being unpatentable over Kato in view of Rhoads. The Examiner 
correctly notes that Kato does not disclose encoding hyperlink information into 
the augmented reality video and accessing information about the corresponding 
product in the rendered three dimensional form from the augmented reality video. 
The Examiner contends that Rhoads discloses that a watermark may carry 
information or programmable actions or links to external information, such as 
retrieval and output of information stored elsewhere in a database, website, etc. 
The Examiner contends that it would have been obvious to one skilled in the art 
to modify Kato's system by embedding hyperlink information into 3D rendered 
objects when selected. Applicants respectfully traverse the rejection. 

As discussed above, Kato does not teach or disclose the creation of an 
augmented reality video in which a 3D image of an object is augmented into a 
video stream. Rhoads discloses using watermarks in video signals to associate 
video objects in a video sequence with object specific actions or information. 
Applicants respectfully submit that adding a watermark as is disclosed in Rhoads 
to the Kato system would not teach or disclose Applicants' invention. Applicant's 
invention is directed to the editing of an augmented reality video in which a 3D 
object is augmented into a video stream. Applicants respectfully submit that 
neither Kato nor Rhoads, whether taken alone or in combination, teach or 
disclose Applicants' invention as claimed. Applicants request that the rejection of 
claims 50, 51, 58 and 59 under 35 U.S.C. § 103 (a) be withdrawn. 
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Conclusion 

Applicants respectfully submit that claims 46, 48, 50-54 and 56, 58-62, as 
amended, are in condition for allowance and request that a timely Notice of 
Allowance be issued in this case. The Examiner is invited to contact the 
undersigned should he have any questions in this matter. 



Respectfully submitted, 



Michele L. Conover 
Reg. No. 34,962 
Attorney for Applicant 



Date: May 18. 2007 
Siemens Corporation 
Intellectual Property Department 
1 70 Wood Avenue South 
Iselin, New Jersey 08830 
(732) 321-3013 
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